\documentclass{article}

% PDF Layout
\usepackage[a4paper]{geometry}
\usepackage[T1]{fontenc}
\usepackage[scaled]{helvet}
\usepackage{verbatim,listings,fourier,courier}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage[colorlinks=true,linkcolor=Blue]{hyperref}
\usepackage[fleqn]{amsmath}

% variable def
\newcommand\pardem{\href{http://www.pardem.eu/}{\textsc{\sffamily\bfseries{\textcolor{Blue}{par}\textcolor{BrickRed}{dem}}}}}
\newcommand\BibTeX{\textsc{Bib}\TeX{}}
\newcommand\citeulike{\href{http://http://www.citeulike.org/}{{\sffamily\bfseries\textcolor{Blue}{cite}\textcolor{OliveGreen}{u}\textcolor{Blue}{like}}}}

% command def
\newcommand\Java{\textsc{Java}}
\newcommand\TAG[1]{\texttt{\color{ForestGreen}#1}}
\newcommand\field[1]{\texttt{\color{RoyalPurple}#1}}
\newcommand\class[1]{\texttt{\bfseries\color{Bittersweet}#1}}
\newcommand\jclass[1]{\texttt{\bfseries\slshape\color{WildStrawberry}#1}}

% title
\title{Updating and Cleaning the \citeulike{} Literature Database for the \pardem{} group}
\author{M.E. Grootens\\\small\href{mailto:m.e.grootens@student.utwente.nl}{\texttt{m.e.grootens@student.utwente.nl}}}

\begin{document}

	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\maketitle
	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\begin{abstract}
		The \citeulike{} literature database for the \pardem{} group is to be cleaned up. This includes replacing or deleting tags (keywords), creating new \TeX{} labels following a fixed format and identifying missing information. This first stage uses the \BibTeX{} export of the complete database.
		
		\begin{itemize}
			
			\item The tags (keywords) present in the current database are sorted; i.e. two lists were created:
				\begin{enumerate}
					\item a list of nonsense tags (tags that are to be deleted), and
					\item a list of sensible tags (tags that are allowed, possibly after renaming them, e.g. by correcting typo's).
				\end{enumerate}
				See Section \ref{sec:tag_cleaning} on the criteria and renaming process.
			
			\item The desired label format is \text{\ttfamily{}[author\_surname]\_[year]\_[first\_word\_in\_title]}.
			
			\item Twelve different reference types---e.g. \class{article}, \class{book}, etc.---are present in the database. The \emph{obligatory} fields---e.g. \field{author}, \field{title}, etc.---are determined per class. See Section \ref{sec:completing}.
			
		\end{itemize}
		
		A \Java{} program was written that subsequently carries out the following tasks:
		\begin{enumerate}
			\item Reads both lists of tags;
			\item Reads the complete \BibTeX{} library;
			\item Identifies obligatory fields that are missing and sets them to ``\texttt{EMPTY}'' (e.g. if no \field{title} is supplied for an \class{article}, set \field{title}\texttt{ = \{EMPTY\}}. Field entries of the form ``\texttt{XXX}'' are replaced by ``\texttt{EMPTY}''.
			\item Fixes the tags according to both lists (i.e. deleting or replacing wrong tags);
			\item Adds a tag \TAG{INCOMPLETE} to all references that have obligatory fields set to \texttt{EMPTY}.
			\item Adds a tag \TAG{INCOMPLETE\_W\_DOI} to all references that have obligatory fields set to \texttt{EMPTY}, but of which the \texttt{doi} is present.
			\item Creates new and unique labels according to the desired format.
			\item Finally writes the new database to multiple \BibTeX{} files of at most 1500 references, so that they can be (re)uploaded to \citeulike.
		\end{enumerate}
		See Section \ref{sec:description_code} on a more detailed description of the program.
		
		Currently the database contains 5947 references, 2968 of which are considered incomplete. Of these 2968 incomplete references, 468 do have a \texttt{doi}, so that the missing information can easily be found. The resulting output files can be uploaded to \citeulike{} and yield no \BibTeX{} errors; the syntax therefore is correct. The in- and output files can be found in the \texttt{dbase} folder of the project root---test database files are present to show specific behaviour.
		
		The next stage is to replace the old database on \citeulike{} and to supply the missing information; this is manual work.
		
	\end{abstract}
	
	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\clearpage{}
	\setcounter{tocdepth}{3}
	\tableofcontents{}
	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\section{To do list}
		
		The to do list is based on the discussion with S. Luding and M. Robinson (May 9) and the email correspondence between M. Robinson, S. Luding and R. Fransen.
		
		\begin{description}
			
			\item[Cleaning up tags]
				
				The tags are used to search the database, but there are many duplicates and meaningless tags. The database needs to be cleaned up. \emph{See section \ref{sec:tag_cleaning} on what was done.}
			
			\item[Importing Granular Matter Journal]
				
				The entire \href{http://www.springer.com/materials/journal/10035}{Granular Matter Journal} is to be included. A collection should be available as a \BibTeX{} file.
			
			\item[Completing citations]
				
				Filling in missing fields in the citations. \emph{See Section \ref{sec:completing} on completeness.}
			
			\item[Update instructions]
				
				Instructions for the use of the CiteULike database can be found at the \pardem{} website. These need to be updated whenever unclear or incomplete. Additional information should be provided on
				
				\begin{itemize}
					\item How to join the \pardem{} group;
					\item How to import/export to \BibTeX{}/Endnote;
					\item How to add individual citations;
					\item Info on the \BibTeX ID format;
					\item Info on the Cite-U-Like RSS feed;
				\end{itemize}
			
			\item[Reformat \BibTeX{} labels]
				
				All labels should be renamed to the format
				\[
					\text{\ttfamily{}[author\_surname]\_[year]\_[first\_word\_in\_title]}.
				\]
				
			
			
			
		\end{description}
	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\section{Cleaning up tags}
	\label{sec:tag_cleaning}
		
		\subsection{Sorting the tags}
			
			Each tag is to be manually reviewed. This might involve looking up its meaning in a dictionary, checking the meaning of abbreviations, or checking to what kind of citations the tag refers. It then is decided if a tag makes sense or not (would/should one use this word to find an article?); this is quite a subjective job.
			
			\subsubsection{Non-sense tags}
				
				\begin{itemize}
					
					\item Words that seem to not have a tag-like meaning (e.g. \TAG{example}, \TAG{nice} or \TAG{consensus}).
					
					\item Adjectives without explicit meaning (e.g. \TAG{easy} does not make sense, but \TAG{supersonic} does.).
					
					\item The author of the citation.
										
					\item \TAG{the}, \TAG{a}, \TAG{an}, \ldots
					
				\end{itemize}
				
				Tags that point to very few articles are more easily classified as non-sense.
			
			\subsubsection{Sensible tags}
				
				A sensible tag should be
				\begin{itemize}
					\item preferably a noun (e.g. \TAG{dislocation}, or \TAG{damping}). But also
					\item abbreviations (e.g. \TAG{FEM} or \TAG{MD}),
					\item adjectives with a clear meaning (e.g. \TAG{dielectric}) or,
					\item names of persons (e.g. \TAG{Euler}).
				\end{itemize}
			
			\subsubsection{Renaming sensible tags}
				
				The number of tags should be significantly reduced. This can be done by renaming similar tags (e.g. \TAG{turbulent}, \TAG{turbulentlike} and \TAG{turbulance} all can be renamed to just \TAG{turbulance}). Furthermore:
				
				\begin{itemize}
					\item Nouns should be singular (\TAG{crystal} instead of \TAG{crystals});
					\item Adjectives that can be renamed to a noun should be (e.g. \TAG{biology} instead of \TAG{biological});
					
					\item Tags should not contain dashes or underscores (e.g. \TAG{dragline} instead of \TAG{drag-line} or \TAG{drag\_line});
					
					\item Some tags need to be split up (e.g. \TAG{euler-lagrange} becomes \TAG{euler} and \TAG{lagrange});
					
					\item Tags should be spelled correctly; 
				\end{itemize}
				
	
	\section{Completing citations}
	\label{sec:completing}	
		
		All citations should be checked for \emph{completeness}. The needed entries depend on the class of the citation. An instance of a class is considered to be complete if it has \emph{at least} its required fields filled in. All classes in the \BibTeX{} file and their required fields are listed below. An empty field will be marked by setting it to \texttt{EMPTY}. It is desired that this information is completed.
		
		\begin{description}
			\item[\class{article}:] 
				\field{author},
				\field{title},
				\field{journal},
				\field{year}
			;
			\item[\class{book}:] 
				\field{author} \texttt{||} \field{editor},
				\field{title},
				\field{publisher},
				\field{year}
			;
			\item[\class{incollection}:] 
				\field{author},
				\field{title},
				\field{booktitle},
				\field{publisher},
				\field{year}
			;
			\item[\class{inproceedings}:] 
				\field{author},
				\field{title},
				\field{booktitle},
				\field{year}
			;
			\item[\class{mastersthesis}:] 
				\field{author},
				\field{title},
				\field{school},
				\field{year}
			;
			\item[\class{misc}:]
				
			;
			\item[\class{phdthesis}:] 
				\field{author},
				\field{title},
				\field{school},
				\field{year}
			;
			\item[\class{proceedings}:] 
				\field{title},
				\field{year}
			;
			\item[\class{techreport}:] 
				\field{author},
				\field{title},
				\field{institution},
				\field{year}
			;
			\item[\class{unpublished}:]
				 \field{author},
				\field{title},
				\field{comment}
			;
			\item[\class{inbook}:]
				\field{author} \texttt{||} \field{editor},
				\field{title},
				\field{chapter} \texttt{||} \field{pages},
				\field{publisher},
				\field{year}
			;
			
			\item[\class{electronic}:]
				\field{author},
				\field{title},
				\field{year}
			;
		\end{description}




	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%	
	\section{Description of the program}
	\label{sec:description_code}


		\begin{table}[b]
			
			\centering
			
			\caption{Project folder structure.}
			\label{tab:folder_structure}
				
			\begin{tabular}{ll}
				\hline\hline
				\texttt{\textbf{core}}	&	contains the \Java{} class defintions\\
				\texttt{\textbf{dbase}}	&	contains the \BibTeX{} databases\\
				\texttt{\textbf{tags}}	&	contains the lists of (non)sense tags\\
				\texttt{\textbf{test	}}	&	contains \Java{} test files\\
				\hline\hline
			\end{tabular}
			
		\end{table}

	
		A short description of what was made and what happens now follows. The structure of the project folder can be found in Table \ref{tab:folder_structure}. The \textsc{change-log} can be found in Section \ref{sec:change_log}.
	
		\subsection{Object oriented implementation}
			
			To give a basic idea of what was made\ldots
			
			\begin{enumerate}
			
				\item	The tags (keywords in BibTeX language) are implemented in a \Java{} \jclass{Tag} class. A \jclass{Tag} contains the original tag name and the name that should be printed. In the case of \TAG{cohesive\_creep} we find the original name to be \TAG{cohesive\_creep} and the name to be printed \TAG{cohesion, creep}.
				
				\item	\BibTeX{} fields are implemented in a \Java{} class \jclass{BibField}. They contain the field name and field entry. e.g. \texttt{title = {foo}} yields a BibField with name \field{title} and entry \texttt{foo}.
				
				\item	There are 12 types of ``\BibTeX{} reference classes'' (\class{article}, \class{proceedings}, ...). A reference is called an entry (of the database, DB); it is implemented in \Java{} as an \jclass{Entry} class. All other entry types are subclasses, e.g. \jclass{Article} and \jclass{Proceedings} all \emph{extend} Entry.
					
				\item	All subclasses have obligatory fields, according to Section \ref{sec:completing}. When an object is instantiated, all its obligatory fields are set to \texttt{EMPTY}. 
				
				\item	There is a bunch of other classes to help manage all objects and to contain some static functions, e.g. for parsing the \BibTeX{} contents.
			
			\end{enumerate}
		
		
		\subsection{Process}
		
		To give a basic idea of what happens\ldots
		
			\begin{enumerate}
				\item	Read the list of sense tags. Store the tags as \Java{} \jclass{Tag} objects such that they can consist of the original tag name (as present in the current DB) and the tag name(s) to which the tag should be renamed. (e.g. \TAG{cohesive\_creep} will be printed as \TAG{cohesion} \emph{and} \TAG{creep}). See the list of tags.
				
				\item	Read the list of nonsense tags.
				
				\item	Read the DB file with \BibTeX{} entries:
				
					\begin{enumerate}
				
						\item Find the \texttt{@}-symbol and conclude that an entry is starting. Read what follows, e.g. \class{article} or \class{proceedings} and based on the type create a new \Java{} object of the right subclass, e.g. \jclass{Article} or \jclass{Proceedings}. These objects already contain the empty fields that are obligatory for the class, but without info, i.e. \field{title}\texttt{ = \{EMPTY\}} etc.
					
						\item Read the \BibTeX{} lines---these should be \BibTeX{} fields---and split them into two pieces: the field name and the field entry. If the field entry consists of multiple lines (can be seen based on the bracket balance), read and add them until a complete field is obtained.
					
						\item Then peel off the outer curly brackets of the field entry, such that only the contents are obtained. (i.e. \field{title}\texttt{ = \{\{foo\} and \{bar.\}\}} is split into the field name \texttt{title} and the entry \texttt{\{foo\} and \{bar\}}.) Create a \Java{} \jclass{BibField} object from these two \jclass{String}s; the \jclass{BibField} will not accept \texttt{xxx} as entry, if this is found, it is set to \texttt{EMPTY}.
					
						\item Add this \jclass{BibField} object to the current \jclass{Entry} object (e.g. the \jclass{Article} or \jclass{Book}). If the field name is one of the obligatory fields, put the entry in the field. If it is an additional field, e.g. \field{comment}\texttt{ = {\ldots}}, then add it to the list of extra fields.
					
						\item Read the closing bracket and conclude that the \BibTeX{} entry is finished; we have now obtained a \Java{} object that represents the reference: it contains all \BibTeX{} fields from the database and obligatory fields that were not filled in, or were marked \texttt{XXX} are set to \texttt{EMPTY}. Note that the labels were neglected; we'll create new ones later.
					
						\item Add the \jclass{Entry} object to a list (i.e. a \jclass{DataBase} object) and start reading the next one, until all entries in the DB are contained in the \Java{} \jclass{DataBase}.
					
					\end{enumerate}
				
				\item	Fix the tags
				
					\begin{enumerate}
						\item For each \jclass{Entry}, read the tags (which are \jclass{String}s) from the \jclass{BibField} that is named \texttt{keywords}.
					
						\item If the tag is contained within the list of nonsense tags, do nothing (i.e. drop the tag).
					
						\item If the tag is represented as a \jclass{Tag} object in the list of sensible tags, then add this \jclass{Tag} to the \jclass{Entry}. (So, if we read \TAG{cohesive\_creep} from the \jclass{BibFiel}d with name \field{keywords}, we add a \jclass{Tag} object to the \jclass{Entry} that will be printed as \TAG{cohesion, creep}, since this was stated in the list of sensible tags.)
					
						\item If the tag is unknown in both lists, allow for its existence and create and add a new \jclass{Tag} object for this unknown tag; a notion will be shown on the command line.
					
						\item Rename the \jclass{BibField} that was called \texttt{keywords} to \texttt{old\_keywords}, so that the new keywords will be used, instead of the old ones.
					
					\end{enumerate}
					
				\item	Create new labels.
				
					\begin{enumerate}
						\item For each \jclass{Entry}, obtain author, year and title. If no author is present (i.e. \field{author}\texttt{ = {EMPTY}}, use the \field{editor} if present (this is the case for some proceedings). Otherwise just use \texttt{EMPTY}.
					
						\item Extract the first word from the title and author.
					
						\item Build the new \jclass{Entry} label: \texttt{author\_year\_name}.
					
						\item Ensure the uniqueness of the label by comparing it to the list of earlier created labels; if a non-unique label was created, append a number, \texttt{++ii}, until a unique label is obtained. Then add the label to the \jclass{Entry}.
					
					\end{enumerate}
				
				\item	Add a \jclass{Tag} called \TAG{INCOMPLETE} to all \jclass{Entry}s that have \jclass{BibField}s with \texttt{EMPTY} as entry; this way one can easily display all incomplete entries in \citeulike{}.
				
				\item	Add a \jclass{Tag} called \TAG{INCOMPLETE\_W\_DOI} to all \jclass{Entry}s that have \jclass{BibField}s with \texttt{EMPTY} as entry, but of which the \texttt{doi} is known.
				
				\item	Write the \jclass{DataBase} to disk. The \BibTeX{} file is split into files of at most 1500 references; otherwise the BibTeX version on the \citeulike{} server will run out of memory, or the session will be terminated because uploading takes too long. The format is as follows:
				
				
				
				\begin{verbatim}
					@entrytype{label,
					
					    obligatory_field1 = {...},
					    obligatory_field1 = {...},
					    ...
						
					    extra_field1 = {...},
					    extra_field2 = {...},
					    ...
						
					    keys = {
					        key1a, key1b,
					        key2,
					        ...
					    },
					
					}
				\end{verbatim}
				
				
				
				\item	Finished! The database now is cleaned and split up into parts that can be uploaded to \citeulike{}. We do a quick check for double \citeulike{} ID's: if doubles are found there might be a bug in the code, since these ID's should be unique.
			
		\end{enumerate}
		
		\subsection{Compiling and running}
			
			The code can be compiled from the command line as follows (\texttt{\$} denotes the prompt):
			\[
				\texttt{\$ javac dbcleaner/CleanDB.java},
			\]
			ones \texttt{PWD} should be one level higher than the project root.
			
			The program can be run by providing the \BibTeX{} file as argument, that is e.g.
			\[
				\texttt{\$ java dbcleaner/CleanDB dbcleaner/dbase/group-13900.bib }
			\]
			In case of the \pardem{} file of the 22nd of august, the prompt would give the following output:
			\begin{verbatim}
				Reading tags...
				Reading database...
				Fixing database tags...
				    Deleting and replacing tags...
				    Tagging incompletes...
				        Num tagged: 2968
				    Tagging incompletes with doi...
				        Num tagged: 468
				Creating new TeX labels...
				Writing database...
				Checking for double CiteULike ID's...
				    No double ID's found.
				Done.
			\end{verbatim}
			Yielding the following files in \texttt{dbcleaner/dbase/}
			\begin{verbatim}
				group-13900_cleaned_08.22_18.01.52_part_1.bib
				group-13900_cleaned_08.22_18.01.52_part_2.bib
				group-13900_cleaned_08.22_18.01.52_part_3.bib
				group-13900_cleaned_08.22_18.01.52_part_4.bib
			\end{verbatim}
			
			

	
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	%% = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = %%
	\section{\textsc{Change-Log}}
	\label{sec:change_log}
		
		\begin{description}
			
			\item[00/07/2012]
				
				General set-up.
			
			\item[02/08/2012]
				
				Changed the required fields according to the email from Martin. Fields are also updated in the PDF.
				
				The \class{book} class now has \field{author} by default as required field. However, when no \field{author} is present, but an \field{editor} is, the required field is changed from \field{author} to \field{editor}.
			
			\item[22/08/2012]
				
				The \class{inbook} class now takes either an \field{author} or \field{editor} as obligatory and either \field{chapter} or \field{pages}.
				
				Added more comments, added the functionality for tagging incomplete entries that do have a \texttt{doi} present. 
				
				Moved the reader for the database file to a helper class.
			
		\end{description}
		

\end{document}